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ABSTRACT 



This paper dTscusses the relevance of, research on- passage 
structur/e and processing for/ the ddsign of reading achieve- 

ment tept^, T^o types of test co^vte^nt 'validity ~are distft\-" 

*- L % ' " • . • ** ' * "* 

guisheq and the relevancTe of psycholinguistifer reseTard} to 

^ • ] . ■ . . ' • * / k *: v 

them is examined. These- taG 4ypes -are the validity accruing 

from, appropriate select io t n, of skills "to be tested, and the 

yalidifty based on relevanc'e ^f -items 'to the selected skills. 

Key linguistic and. psychological terms are defined and 

I . * 

several specific/ reading models are discussed. Their poten-* 

tial contributions .to design - of reading achieyement tests 

are assessed. Finally, a*bri£f passage^ is analyzed*, 1 and 

some practical applications of g the theory of .text structure 

and processing artf examined. , I \ 
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• . INTRODUCTION *** , 

4 

Sternberg .(1981) advocated the application of cognitive 

and information processing psychology to psychometric test- 
« • 

ing. Although tjie vocabulary of cognitive psychology is a 
novelty ii* reading achievement 'testing, it c§n and should be 
a useful adjunct. The geheral aim*of this paper is to dis- 
cuss ways in which psycholinguistic and educational research 
tan be applied to the design of reading achievefoqnt tests. 
A central consideration in test construction is content 
validity. Typically, one' of the main objectives of a read- 

ing achievement test is to, measure how well a* person is able 

■> 

to understand a passage. One challenge in test construction 
is to operationalize the term* "understand" in the foregoing 1 
sentence. The intent here< is not to carry out a complete/ 
and exhaustive * operationalization .of this term.* This 
requires taking . into account the particular circumstancp^ 
and goals associated* . with each test. father, the intent 
here is to ^stimulate awareness of research that appears par- 

* * . ' " t * 

ticularly promising, to illustrate relevant. issues, - and to 

* s " • . - I 

'describe a concrete application of the theory to test 



i 



design. "~ . t 



"1 • 

4 



*• Frase (1975, p. 4) suggests that -what is learnedly read- 

. : « • ..••]- 

ing is "the result of. an interaction between what the reader , 



attempts to do and Ahe constraints that the stimulus 
materials place on those adaptive strategies." This defini- 
tion focuses attentidn on the passage ftself and on the cog- 
nitive*processes incomprehension'. Dne limitation of early 
psycholinguistie research was the lack of a theoretically 
based yet practical way of describing the meaning of a pas- 
sage. More recently there has been a trend toward research 
which is based on explicit theories of passage structure. 
'(Frederikseri,* 1975a, 1975b; Meyer, , 197^, 1975; Kintsch and 
van Dijk, 1977;. Grimes, 1975; Halliday and Hasan, A976; .van 

* n, 

"6 1 

Dijk, 1978; and brothers, 1979). This research has begun to 
interest educators. Lucas and'McConkie (1980) parsed pas- 
sages info a propositional network representation in order 
to. identify propositions which are relevant to test items 
and describe the relationship of the proposition to the 
-.item. This is ih the -tradition of^ content standard testing 
discussed by Ebel (1962) and' Bormuth (1970). A traditional 

problem with readability indices h£rS been their lack of sen- 

' . V* 

sitivity to passage content.. Kiivtsch fc£979) has described a 



readability formula which takes into account reader infer- 

- " ( * 
ences and the proposit ional structure of passages. . Crothers 

(1972), ' Gentner (1976)* •Kintsch, (1974) and Me^er and 

McConkie" (1973) provide empirical support for a relationship 

between comprehension and memory and thq hierarchical level 

of information ' in the passage. , . Brown, - Campione and Day 

' ] * ] "'' ' 

(1981) '\ describe *the use of summarization rules ^as a check 

' ! \ 

for comprehension and detention of information. 



x These -studies support tlje concept of reading as inter- 
action between passage structure' , and cognitive processing. 

- ■ 

^It is worthwhile, to consider the implications, of i^his per- 
spective for the^design of reading achievement tests. The 
next section of the paper discusses ways in which theory can 
be helpful in selection and definition of skills to be, 
assessed" by a test. . /' 

r ■ * . /j 

Content validity. Traditionally, • M to demonstrate the 

/ / - 

content va\4dity of a set of test scores, one mjist show that 

* the behaviors demonstrated in testing consti repre- 

* / 
sentativejusample of the behaviors to be /exhibited 'in a 

• ' • / 

desired performance domain." (Standards, ' lp74) . Test item 

specifications are designed to define ifnd describe* the* 

skills that the^ tdst is intended to assess*. In particular, 

a set of item specifications can be used to examine perform- 

ance in particular skill areas. On the basis of this exami- 

nation educational resources can be allocated Rationally to 

ar.&as of need. < In order to make these kinds of decisions 

with confidence the test must possess contfent validity. ' Two 

aspects of content validity can be distinguished here.. On 

the one. hand, 3 definition of the appropriate performance 

domain can b*e obtained from textbooks in * common use, 

descriptions of scope and s^fluence, and other Relevant docu-. 

ments. The second aspect of validity requires that the 

items assessing each selected skilT are in r fac£ relevant to 

* * 

that skill. For example, a skijll cofiflnonly tautjht is infer-. 



ring, the main idea of a passage. A brief operational 
definition of ihis skill area might be the following: "given 
a passage, the student will identify the primary topic of 
the passage.' Suppose a* multiple choice item is^ needed. The 
task of the item writer is to construct an Item requiring 
the student to pick the theme from several options, one of 
which expresses the main idea, and the others of whi^h do 
not. The operational definition contains two problematic 
terms, "identify" and "primary topic 1 *. "Identification is a 
cognitive process which involves inference, "Topic" is a 

literary^linguistic term which is precisely' def ineable 1 in 

- •„ / 

the content of a theory "of passage structure. Typically, 
the item writer receives little guidance in defining these 
terms and must proceed on the basis of intuition* Undenia- 
bly, there are various intuitive conteptions of this partic- 
ular skill, of wiiat constitute^ the topic of a passage, and 

\ 

how the topic isi inferred. *Many of these intuitive concep- 
tions may possess more than a little validity. Yet/ Khen 
one considers, the enormous amount of resources devoted to 
testing* and the importance of the decisions jnade on the 

basis of testing outcomes, it is apparent that more than 

t < 

just- intuitive conceptions are needed. Psycholinguistics 
/ 

provides theoretically 'based and empirically^ tested proce- 

dures for defining such skills. 

* • 

Definitions. \li ? is nserful to define briefly some key 
terms. The meaning of a passage can be, represented in terms 



of a network of propositions* and" connectives. Propositions 

consist of a predicate, represented in the passage fiy a verb 

» * 

or an adjective, and one* or more arguments. Propositions 

•are uaderlying units of meaning, and should no't bfe confused 

with the passage surface structure, even though words from a 

* 

passage can be used conveniently to represent the elements 
of a proposition. For instance, a sentence, "John gave Mary 
a hug" can be represented as a proposition by (GIVE: JOHN 
MARY HUG).. In this; case the predicate is "GIVE", and the 
arguments; presented in order of agent, recipient and* objetjt 
are "JOHN", "MARY", and "HUG". This type of analysis, has 
its roots in Fillmore's (1968) theory of case grammar? 

The set of .connectives • includes conjunctions, such as 

"and", "or", "if-then", and so on. Other types of conniec- 
\ * 

tives are expressions of causality, spatial and temporal 

4- 

contiguityj and reference. 'Meyer (1975) and Crothera (1979) 
have noted that connectives* help to establish the ordination 
of propositions. For in'stance, the connective "OR" can be 
'used to coordinate two or more propositions. By contrast, f 
the connective "WHEN" establishes one proposition as super- 
ordinate, and others as subordinate. 

Propositions can be explicit or they may^be inferrq/d. 

♦ 

Inferences are drawn in order to connec£ otherwise isolated 

• J ' 

propositions to the passage structure. Authors often assume 
-background knowledge on the part of the reader, and do not 



include information that appgars redundant,* or is commonly 

presupposed. In order to grasp the 'author's intent this 

♦ 

information must be inferred by the reader. 
« 

Th6 TDicrostructUre of a passage 'consists o*f the relation- 

ships between. the elements of individual, propositions, and 

of the relationships between propbsitions (van Dijk, 1977). 

« 

Text macrostructure' refers to the ordination of proposi- 

tioris. Certain 5 - propositions are* more superordinate thai\ 

• * * » 

others. The ofdinat^dn of propositions establishes a hier- 
archy. The description of the levels of this hierarchy is a* 
description of- the passage ifiacrostructure. Superordinate 
.propositions usually occur in a summary or • abstract of the 
passage. Propositions which are' subordinate represent pas- 
sage detail. 

I 

\ * 

Kintsch (1977) has remarked t^at the reader must be able 

* 'o 
to integrate the new information presented in a passage with 

/ " ' - \ 

the old information. The coherence of the reader 's- repre- 
sentation depends on this synthfekis of information. Coher- 

n > . 

ence requires the c*bs'6nce of log/Leal contradictions, a con- 

sislent causal, spatial andLi^mporal ordering of -events and 

objects, "and 'the consistent use of referring expressions. A 

requirement for coherence oi; macrostructure, is that the pas- 

* 

sage have some definite theme or topic. 

Conventionally written passages possess a schematic 

structure or schema. The 'concept of schema (Minsky, 1975) 

i 



:10 



rpfers to^a traditional organization of the passage. For 
instance, manf stories are ocganiczecl 4 n *o a "setting, * com- 
plication, resolution 'and evaluatiorf*' pattern. The. setting 
•typically describes the physical circumstances of the st<jry_ 
and introduces the characters, , The complication i& genera- 
ally, a problem fio be solved. The reTsofution is the solution 
of the problem," and the evaluation is the moral or point M 

the story. Research reports are another i;ype .of passage* 

i # 

- ' • * * N • * 

with a wel.1 defined schema. These usually contain .an 

abstract, literature review, statement, of hypotheses,' method 

section, results and discussion^ Fetler' (1979) ha& dis- 
*♦ >. * 

i 

cussed schemata in question and answer dialogues. Pre$uma- 
bly, experienped readers tend to expect schematic organiza- 
tion, and use it to guide their * processing of the passage* • 



Models of reading.* What fo'llows - -is a brief discyssion 

«.>.* s u ' '• 

of t'hrejb contemporary process models of reading. The pur- 



pose of this section is to describe concepts of cognitive 
processing ancl passage Structure. The models of Frase 
(1975) and Rothkopf (1976) illustrate the range of cognitive 
processes used in reading. ' The model of 'Kintsch and van 
Dijk (1978) illustrates the use of detailed linguistic 
descriptions of passage structure. The process model 
approach clearly distinguishes between the material to'be 

read, the processes o,f comprehension, and the memorial con- 

W 

sequences of the processing. 



Frase frames his model in terms of four- levels of proc- 
essing. The results of earlier levels can Effect later 

processing. The reader's activities are goal .directed and 

\ 9 

the' performance or goal feet can* be influenced by explicit . 

learning goals. x Level I processing involves establishing 

the performance se£. Encoding of orthographic, ' syntactic- or % 

semantic aspects of the passage >is level II processing. . 

Performance set and encoding determine what is available to * 

Jiiflher level processes. Rehearsal r and integration of . 

encode information are level III processes. At this "stage 

> * <t ' « 

information may be input to 'long term memory. ^ Level IV 

processes, the* retr i§3|bl anti generation af information oper- 

u ' * 

atp* independently of the ear lier* processes. 

Rothkopf distinguishes, between nominal stimuli»or physi- 
cal pas # sages> and ef fective^stimuli or ^usable repesentations 
of passages* reconstructed by-. the reader. ^ Primary mathema- 

genie processes take nominal stimuli as input. These proo 

'0 /' , & . * 

esses includfe eye* movements* translation of the input into 

' ' ' * • 

acoustic-motor format, and syntactic analysis.- Secondary 

• / 

mathemfigenic processing involves operating 6n usable repre- 

s&tations b£ collation, integration and elaboration of * 

» • t r * 

* * •* 

information. ^ Rehearsal regulates the transfer o,f. .usable 

representations into long term memory.. * -A passage, 'can be ^ 

' . . . • ' - , ^ % ; 

processed in different t ways depending on task demands. This 

may invplve selective processing of*different parts of the 

passage, and differences in the extent to which certain 

i * 
information* is. rehearsed or" elaborated. 



•12 . ; 



Rothkopf and Frase describe in detail the^ translation of 
printed- sentences int<£^smory # representations. TtteYeading 
processes in their models are of two kinds. Either they 
describe the encoding of sentence meaning, pr they describe 
comprehension processes which operatfe on the encoded mean- 
ing. The processing is guided' in accordance with the 
reader's particular goal orientation. As a result it is 

possible to describe the effects of various learning strate- 

f 

gies adopted by the reader and of learning instructions. 
The processing described by Frase and Rothkopf is linear in 
the sense that their models operate serially on one sentence 
at a time. The-linea^ approach can be expanded, however, to 
take into ^acqount'the relationships between many different 

propositions in?^ passage's, structure. 

. • ' * " # 

Kintsch and van Dijk assume that, the meaning of a passage 

can be represented in terms of ^nicrostruqtuYe and macros- 
tructure. T&jree types of processes are distinguished. 
First, propositions are organised into a^ coherent structure. 
A second set of processes reduces' this set of organized 
'information, into a summary^ Finally, a tfiird set of opera- 
tions governs* recall 'of stored inforftat-ion. Encoding* o£ 
propositions is not part of the model 8 ? although similar 
processes are^described by Frase and Rdthkopf and have ^been 
studied by Anderson anc^Bower (197SV and Kintsch (1974)^ 
Propositions are processed in groups of five to nine in the 
order of their appearance in the passage. , The processing of. 



. . 10 

each group of propositions is called a cyc^le. During each 

ii ^ » 

cycle some of the propositions ar£ stored in short term mem- 
ory, and are available-ior connecting with the next set of 
input propositions. Processing in . short term memory 
'involves a finite probability of storage in long term mem- 
ory. 

The model produces a coherent network of propositions. 
The nodes of the memory network are. individual propositions, 
and the linesv^are shared references. The organization of 
the network is determinfec} by the chbice of propositions left 
in short term memory beKfcdren cycles. JThere are two strate- 
gies ior selecting these propositions. - First, those propo- 
sitions should be selected which are important dn the sense 
of being connected to many other propositions. The secoftd 
principle is that more recently processed propositions are 
more Tikely to be selected. 

A summary of the passage is obtained from the macrostruc- 

• • 

ture. Propositions that are redundant or irrelevant may be 
deleted 'or generalized,, smd new propositions may be infer- 
red. The deletion, generalization and inference operations* 
are applied in cycles with increasingly strict criteria for 

9 \ 

relevance. Products are a summary, abstract, theme ^and main 

idea. » \ 

* \ 

Applications in test design. How ntight these theories\be 
useful in the design of reading iests? Applications arfe 



.' 11 

suggested by^ the distinction of twp componehts of* co nten t 
validity: Certain skills to be assessed must be selected;, 
•and there must be a§s^ratfc,e that the items in JPSc^ assess 

/fhe' selected skills. Assuring that .the selected Skills are 

f .}» . » 

in f.act assessed, is a matter of careful '* analysis of the 

Jill 

stimulus passages and items". Iv|j$s in the analysis of pas- 
sages, and items that theories of structure- and processing 
are. useful. One advantage of relying on explicit and expe- 
rimentally tested theory is the possibility of systematic 

s v 

classification of items in terms of skills. Analysis of 
passage structure involves the development of an extensive 
typology of types of propositions, connectives,*, inference^ 
referential devices and rhetorical constructions. Explicit 
criteria can be specified for generating summaries, 
abstracts, themes and topics of passages* Generally, given 
a network representation of the meaning of a passage, it is 
possible to locate and describe local or global patterns of 
organization, to tabulate recurting phenomena, t and to 
describe the component propositions in terms of their rela- 
tions to other- pr6postiqns. The various identifiable struc- 
tures and processes involved in comprehension can be objec- 
tively correlated with skills. 

V 

Analysis of passage structure, can be used as an editing 
. procedure to check the coherence of stimulus passages. Pas-- 
sages. .which are not'well written can be expected to confute 

0 

4 

or distract \he reader, and introduce "npise" into the meas- 



; 12 

# * ' j 

urement process* Examples of' phenomena which detract from 

coherence are literal or inferrable conijt-adict ions, the 
presence of propositions which do nob relate to the rest of 
the passage, ambiguities in. ^pati^l, * .temporal ~ or causal 
sequence, ' and references which are not satisfied or are 
vague. Even careful item writers and proof readers % can 
"sometimes- over look these kinds of phenomena.-* A systematic 
analysis of the passages is helpful in avoiding such over- 
sights. * ; % * m 

How the analysis of a passage might proceed is illus- 
trated here. The passage selected for analysis is a short 
narrative. The content, t length and difficulty are typical 
of what one might find in a third grade reading achievement 
test. The body of the passage follows. % 



1 



The wind was icy cold, and black* clouds were cov- 
erijg, the flight of the moon. Bob saw a little 
cave in the rocky mountainside^. He climbed 
between two big .rocks. Suddenly he fell, half 
frozen, to the dry ground inside the cave. \^ He 
tried to catch his breath. Then he heard it - the 
sl<Jw, dee£ br?arthing of a sleeping bear in the 
corner of tfie cave. 



13- 



jfhe- first stage of the analysis involves a decomposition 



7 



of sentences" into gpnropositiQ^s * using methods similar to 
those of^urn^r and Greene (1979). ^The results of Jtftis are 
shown in Table 1. For convenient reference, individual 
propositions are numbered in the column on the left* On the 
rfght, propositions are displayed, enclosed in parentheses. 
Predicates are written first, followed by a CQlon. v Argu- 
meirts are separated by commas. Horizontal lines separate 
groups of propositions corresponding to dif f erreiTt sen- 
' tences. > s - - - 



Insert Table 1 about here.' 



The first six propositions correspond to the- first serf- 
tence, propositions seven through twelve correspontK^o the 

' , *r " W- * \ 

second sentence, and so on. ProposflKbn three illustj 
that entire propositions can be embedded as arguments in^Aew 1 
propositions. *Here 'the predicate is a conjunction, &nd, thf 
arguments are propositions one and four. Proposition twelve 
illustrates this in a slightly different way. Bob^s clim- 
bing action is described in proposition eleven. The location 
of that actio^is described in proposition twelve. 

Figure 1 displays a network representation of the pas- 
sage.- The^tfmfcers correspond to the propositions iir Table 
1. lin€s correspond to connectives between proposi- 

tions. Not'a^l possilbt^ lines are* drawn, but only* those 
-needed to display the ^overall structure 



>. ^Vor i 



hstance, 
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' ' • « \ ' 

based on the relationship of identity, it would he possible 

'. 

to' connect every proposition . containing the element "BOB" 

with every other proposition containing that ' element. 

' * ff* \ . 

Although this wouXd produce a more complete analysis, the 

resulting network would be much less perspicuous, % arid less 

lisefdl for applied educational and psychological purposes. 

The structure of the network is largely determined by the 

proposition list. For instance, the repeated argument, 

"COLD", is' the basi$ for connecting propositions 1 and 2. 

The conjunction in proposition 3 is a "branching point for 

for 1 ^nd-4. * • 



Insert figure 1 about here. 



; Writing test ^ items. The analysis presented so far is 
v£ef ujl for writing and grading certain kinds of test* items. 
For instance> to test the .skill of remembering the gist of a 
passage, a student can be asked to write or select a sum- 
•mary, an 'abstrapt, or eve'n just the theme or topic of the 
passage. An objective way to 'determine the topic of a pas- 

sage is to look' at argument repetition. This can be done 

■*■■> 

either within each schematic section, or overall. Looking 
at* frequencies overall,^ "BOB" is repeated explicit ly -eight* 
times, compared to five times both for "CAVE" and ^BREATH- 
ING", (The "IT" in 24 counts once for "BREATHING". ) If 
implicit repetitions from embedded propositions are included 
in the counts, the tally for "BOB" rises to fourteen. Pre- 
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sumably any description of the topic would take account of 

these f requehtly ; recurring elements. Of course, if the p£s- 

«> * * ♦ 

-sags included a resolution and evaluation the counts might 

% I „. * 

An 'approach which relies more on the meaning of the story 
is to analyze the network representation. The depth of a 

proposition is defined as the number of propositions to the 

> 

left of it in the network. Thus, propositions 3, 7, IV, 15, 
21 and 23 have a depth of zero, propositions 1, 4,. 9 £tc. 
have a depth of one, • and so on. The deeper a proposition 
is, the more likely it represents detail. The more shallow ■ 
the proposition/ the more likely it represents thematic 
material* Thus,^n the introduction, the most salient ppint . 
is that Bob saw a cave, (proposition 7). That the cave j^ls 
little* in a mountainside, and that the mountainside is 
rock?; (propositions 8, 9 and 10) are ancillary facts. A 
summary of the passage would delete ancillary material, 
retaining only what .is mols^salient. A good summary of the 
passage would include that Bob saw a cave, that he cl-imbed 

and fell into it, and that he^Keard the breaching of a bear. 

- *. , v- 

It would not include the details that the ground of the cave 

was dry, or that the bear was in a corner. 

******* i , » . . 

The analysis helps to distinguisjh objectively 'what • is^ 

literally present in the passage from what must be <4pf erred. 

The .analysis in Table 1 is purposely very literal in that it 



16 

only contains what is explicit in the text. However, from 
the setting one can infer that the^ime, is night* and les| 

'"• ' ( 

trivially that Bob climbed into the cave for shelter from 
jtfte icy wind, that the two big rocks are by the mouth of the 
cave, and that Bob had to catch his breath from the exertion 

v. , 

of climbing and falling. Once these inferences are brought 
out, one can much more objectively write items testing fgr 
literal and inferential comprehension. 

Conclusions. The purpose of this paper has been to dis- 

cuss ways .in which psycholinguistic and educational research 

on passage structure y^is be relevant to reading achievement 

tesj^ 'design. The main argument is* that" this research can 

help provide a more objective basis for the writing of pas- 

sageg and items. A concrete example of this, involving the 

> 

determination of a passage summary and^ topic was illus- 
trated. 

• It might be argued that* test construction" is already 
quite difficult enough, qnd should not have to shoulder the 
additional burdens described here. In reply, tjie mpdels of 
Frase, • RothRbpf and Kintsch and van Dijk mafce clear that 
reading is not a^simple process. r To assess the use of read- 
ing skills, both scientific 'descriptions of the skills and 
of^thfi^eading mater ials *are needed. Intuitive conceptions 
oi these can ;serve only up %o a point. Scieptifiq concep- 
tions, grounded in empirical research, wpuld be an improve- 

v • * > - • . 

ment* " ' 
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Figure 1: Network representation of the sample text. 
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